Interpolation B + tree : an efficient file structure with which to store and search large almost static data sets

Many applications generate very large data sets that are too massive for effective manipulation using conventional main memory techniques. One specific application area associated with DNA (DeoxyriboNucleic Acid) research, is the log that tracks samples through the analysis process. Analyses of this kind can easily produce 10,000,000 records in a short time period. Tracking a single sample through the process involves searching this very large log for the set of records storing the unique bar code associated with the sample. In this paper a specialized external memory data structure is proposed that will provide efficient access to the required records. We examine an adaptation of the Pegasus method for finding the root of a continuous function in conjunction with a suitable external memory data structure. We show that applying the Pegasus method together with a modified B tree provides close to optimal I/O efficiency for executing a set of point queries on large log files.